A number of vision-based methods for detecting laser-induced defects on optical components have been implemented to replace the time-consuming manual inspection. While deep-learning-based methods have achieved state-of-the-art performances in many visual recognition tasks, their success often hinges on the availability of a large number of labeled training sets. In this paper, we propose a surface defect detection method based on image segmentation with a U-shaped convolutional network (U-Net). The designed network was trained on paired sets of online and offline images of optics from a large laser facility. We show in our experimental evaluation that our approach can accurately locate laser-induced defects on the optics in real time. The main advantage of the proposed method is that the network can be trained end to end on small samples, without the requirement for manual labeling or manual feature extraction. The approach can be applied to the daily inspection and maintenance of optical components in large laser facilities.